Subspace distillation for continual learning

An ultimate objective in continual learning is to preserve knowledge learned preceding tasks while new tasks. To mitigate forgetting prior knowledge, we propose a novel distillation technique that takes into the account manifold structure of latent/output space neural network achieve this, approximate data up-to its first order, hence benefiting from linear subspaces model and maintain concepts. We demonstrate modeling with provides several intriguing properties, including robustness noise therefore effective for mitigating Catastrophic Forgetting learning. also discuss show how our proposed method can be adopted address both classification segmentation problems. Empirically, observe outperforms various methods on challenging datasets Pascal VOC, Tiny-Imagenet. Furthermore, seamlessly combined existing approaches improve their performances. The codes this article will available at https://github.com/csiro-robotics/SDCL.